Remote Direct Memory Access (RDMA) is a feature that is provided on some communications infrastructures and allows data to be written to, and read from, specific locations in memory without the need for data to be received at a central buffer and copied to the destination addresses under CPU control. The sender of data specifies, in a form understood by the receiver, where the data should be placed at the receiving end. The data might then be placed on the receiving end without having to examine a complex context. The receiver might even delegate the data placement to specialized hardware. When data has been successfully delivered into the receiver's buffers, the receiver must be notified of the completed transfer (usually by some kind of interrupt mechanism).
RDMA may find application for data transfer between storage devices and servers in an IP network. In such case, RDMA allows a network device to read or write remote memory. Control information associated with the data specifies the buffers to be read or written. The remote network card extracts the identity of the buffer and uses DMA to read/write memory directly.
JP020490A2 assigned to FUJITSU LTD., published Jan. 21, 2000 and entitled “Computer having remote procedure calling mechanism or object request broker mechanism, data transfer method and transfer method storage medium” relates to the desirability of shortening the delay of data transfer between computers having object request broker and remote procedure calling mechanisms by directly transferring data from a physical memory area built in a communication source computer to a physical memory area in a communication party side computer. To this end, a remote direct data transfer unit executes remote direct memory access (RDMA) for directly transferring data stored in a physical memory built in its own computer to a physical memory area in a communication party side computer. A system area network connects the computers and is used as a physical communication line to effect data transfer by the remote direct data transfer unit.
U.S. Pat. No. 5,978,865 published Nov. 2, 1999 and assigned to Advanced Micro Devices, Inc. discloses a system for performing DMA transfers where an interrupt request signal is generated based on the value of the last of a plurality of data bits transmitted. A micro-controller is configurable to transfer data to and from one or more asynchronous serial ports (ASPs) using direct memory access (DMA), and having hardware features which cause each ASP to notify the execution unit when a data frame having a last data bit equal to a predetermined value is received. Such hardware features allow the execution unit to determine when complete data packets are received. The serial communication data is transmitted within data frames, wherein each data frame includes multiple data bits transmitted sequentially between a start bit and one or more stop bits. The last data bit of the multiple data bits is transmitted immediately before the one or more stop bits. Each ASP is configurable to generate an ASP interrupt request signal when the value of the last of the multiple data bits is equal to a predetermined value.
U.S. Pat. No. 5,978,865 does not appear to address the division of a data transfer over multiple channels. They describe the workings of a DMA engine, passing data from a communication link to memory. When each individual transfer of a data packet is completed, they generate an interrupt. They do not address a data transfer that has been spread across several channels or that has been divided into multiple data packets, where an interrupt is required only after all of the data packets for a particular data transfer have arrived.
Reference is also made to a working draft by Cisco accessible on http://www.cs.duke.edu/ari/publications/draft-csapuntz-caserdma-00.txt. Particular reference is made to Section 6 entitled “Implementing RDMA”. It is to be noted that this paper, while relevant to the general field of the invention, proposes several solutions to the problem of framing but does not address the problem to whose solution the invention is directed. A complete copy of this paper is being deposited at the Patent Office so as to allow access thereto in the event that Internet access is disabled.
Regardless of the specific application for which RDMA is used, a known problem of RDMA relates to determining when a data transfer has been completed. This is particularly acute when a large data transfer (“transaction”) is broken down into several smaller data transfers (“packets”). The receiver must be informed that the entire transaction has been completed. An RDMA engine may know how much data has been transferred on each packet, and it may also know how much data makes up the entire transaction. The RDMA engine would then have to keep track of how much data has arrived for each pending transaction constituting the complete data transfer, and would generate an interrupt when it has received the total number of bytes that were specified for a particular transaction (after receiving some number of packets).
The problem is compounded when the individual packets constituting the transaction are sent to the RDMA engine via different network fabrics. In this case, no single RDMA engine on the receiving end receives all of the data for a particular transaction, and therefore no single RDMA engine can know when the transaction has completed. It is therefore known to generate an interrupt or callback for each packet on each of the RDMA engines, and compute the total data delivered for the transaction in software. This solution has the undesirable condition that it results in an interrupt being generated for each packet. The receiver is interested in knowing when the entire transaction (comprising all the data packets) has completed, and all the extra interrupts/callbacks for the small data transfers consume resources that could otherwise be used for other purposes.
The same problem obtains when transmitting data having a main header following by multiple data packets, each having its own sub header containing less comprehensive addressing information. The main header contains most of the addresses for directing each subsequent data packet to a specified buffer memory, as well as the combined length of the data in all of the subsequent data packets. Consequently, without the main header, it is not possible to place the subsequent data packet, and it is not easy to determine when the data transfer has completed. There is therefore a serious problem if the main header is lost or arrives garbled.
One proposed solution requires identification of the first packet in the next string so as to identify the arrival of the next main header. If, at this stage, it is not possible to deliver the preceding data packets owing to loss of the previous main header, then the preceding data packets can either be discarded or buffered pending possible recovery of the preceding header.
The requirement to identify the start of the next packet also has several drawbacks. First, if there is no subsequent packet, then it is neither clear whether the received data is complete nor what to do with the data so far received. Secondly, such an approach is suitable only if all the data packets are sent on a single channel since if they are divided among different channels, and some of these channels have finished sending data while data continues to be sent along one or more remaining channels, then so far as those channels are concerned where the data transmission is now complete there is no indication that no more data will be sent. This is similar to the first problem but even more acute since the main header is sent once only on only one channel and so inevitably arrives disassociated from all those data packets sent on different channels.
It would therefore be desirable to provide a solution to this problem that minimizes the number of interrupts in determining when a transaction using RDMA has completed.